The collection, consolidation and general centralized data processing of business data is a necessary pre-condition to the operation of modern data warehouse systems also known as Business Intelligence (BI) systems and or “Big Data” (BD) systems. In order to analyze and or visualize the data, calculate statistics, compare values, identify or generate key data elements and in general provide modern business reports on business data, a centralized database system must incorporate, take in or consume and process data from one or more remote or external business systems. This general data input process is known as an Extract Transform and Load (ETL) process which various database or other software vendors may provide as part of their core SQL database or data warehouse or BI systems. However, some of the key problems in building data warehouses or BI systems in general is identifying and describing which types of data are required to be collected, identifying where the required data is located in various foreign or remote business systems and most importantly extracting and processing the required data from multiple heterogeneous remote data sources, while performing these steps on a recurring and ideally automated basis. Typically, ETL processes are fed from well-known or standardized enterprise type business systems such as ERP, CRM or CMS systems using well known bulk data extraction procedures or even more modern web-services type interfaces which are provided by the external enterprise data source system. This type of architecture works well for large companies and or enterprise type data sources as there is sufficient personnel, technology infrastructure and documented system interfaces or Application Programming Interfaces (APIs) to identify and extract this data which is then fed into the ETL process built by the enterprise IT personnel or vendor supplied systems. Most commonly, these data sources are located within or operate internal to an organization (e.g. within their own private network, or behind an Internet firewall on a common shared network) and or transfer data from Internet based SaaS systems over the public Internet using Service Oriented Architectures (SOA) web-services with published, well-known or even privately negotiated interfaces or APIs to extract data and send it into an ETL system. In the simplest case, an ETL system may use simple methods such as a SQL query for data extraction to a flat file combined with a FTP file transfer to a secure landing site operated by the existing ETL process. These types of systems designs are well known to those with a general level of IT knowledge or those who work for large corporations, and or consult or support data warehouse or BI systems, and or work for BI or data warehousing or ETL type vendors. This type of data processing is in common practice at an enterprise level where sophisticated and custom systems integration tools, data modeling, and software development skills are in abundance along with the resources to create, manage and support such systems including storing vast quantities of data, or “Big Data” (BD) in a system.
However, this well-known ETL design does not work for the majority of businesses today—that is the millions of small and medium (SMB) sized businesses who have neither the enterprise IT skills, personnel or infrastructure (or even budget) to operate such a complex ETL and or BI system. What is needed therefore, and embodied within the present invention, is the ability to operate a similar enterprise like ETL process in order to create an SMB BI or BD system which works across and adapts to a variety of non-enterprise software and data sources, including at thousands or even millions of remote sites in a lights out, automated manner while still allowing the various SMB members to benefit from the business insight gained from BI reports, dashboards, scorecards, tools and or metrics. The general purpose of the SMB BI system is to help business owners, operators, managers and even employees to understand how their specific or local business is performing using a wide range of key business measurements, statistics, computed or qualitative scoring, ranking and identified data values in a variety of reporting formats or types including enhanced visual reports, especially on an individual and or peer group basis.
The difficulty of designing, let alone operating this type of system is significant when one considers that these SMB business owners may own or operate one or more SMB locations but generally do not have detailed database, data warehouse, BI or even general IT skills at their location, let alone multiple remote locations. While they may have selected their LOB systems from commercial off the shelf (COTS) software vendors, or customized it extensively by a VAR or other vendor, or even had them custom built by programmers hired for the project, they usually do not broadly understand the internal operations of these LOB systems. This includes how to get at, define, extract and select specific data sets from a specific system and integrate that data subset into other systems. Thus operating a general purpose ETL system which can dynamically adapt to the wide variety, types and conditions found within various SMB locations requires the specific abilities of the present invention. Without the embodiments of this system design, the successful collection and consolidation of the required remote SMB data can't be provided by the average small or medium business owner, manager or employee, nor can the SMB BI system be utilized to improve their business with having access to “all” of the data in a consolidated, comparative form.
In order to participate in an enterprise type BI system, SMB businesses face a variety of difficulties if not outright obstacles or barriers to success. First, each remote SMB business must provide business data on a frequent if not regular basis which can be time consuming if not prohibitive if it cannot be made available to them in an automated manner, without expensive software customization, programming and or consulting labor. Second, the data provided must be processed from the internal format used by the desired LOB, POS or financial and or accounting systems and transformed into a standardized or normalized form which may be required by the SMB BI system to make an “apples to apples” type comparison across various members. Most likely, the SMB business participant does not understand how to generate this type of reformatted or transformed data from their individual LOB systems, let alone have the ability to produce it on a frequent basis and in a rigorous and consistent manner. More significant to the overall system design, each SMB owner may not be utilizing the same database design, schema or database elements, particularly if they utilize different version levels of the same system from the same vendor. This challenge is made even worse if multiple SMB sites use different types of LOB systems (i.e. inventory, sales, ordering and the like) or systems from different vendors. Thus the ETL process is generally unmanageable, if not outright impossible for individual SMB participants.
Among the millions of SMB businesses in the US, many are based on the franchised business model. Franchising is an approach to doing business whereby the owner of a business concept (i.e., a Franchisor) licenses the business model to an independent business entity (i.e., a Franchisee) to operate under the terms of a well-defined franchise agreement. The Franchisor authorizes the usage of proven trade dress, operating methods and systems as well as business practices within the guidelines set forth in the agreement. The Franchisor provides various support services that may include advertising/marketing/public relations, training, operating/physical facility design assistance, product/menu specifications, material sourcing support, design and production of business operating systems or methods including specification or design of specific Line of Business (LOB) applications, Point of Sales (POS) systems and/or other business support. Such support is generally provided to the Franchisee for a royalty fee, typically calculated as a percentage of gross monthly sales which the Franchisee typically faxes or emails to the Franchisor on a monthly basis. In the US, as of 2005, the franchising business model was utilized in over 70 different industry segments by 2,500 Franchisors with over 900,000 individual franchisee locations generating more than $1 trillion in revenues. Each business utilizes one or more LOB applications which are a set of critical computer applications that are vital to running various aspects of the business, such as production, operations, sales and marketing, accounting, supply chain management, resource planning applications, and the like. LOB applications are usually specialized programs that contain a number of integrated capabilities which tie into and depend upon the features of databases and database management systems to store, retrieve and manage critical transaction and business data. Even with this vast level of support, most individual franchise owners struggle to understand the health of their business, let alone have the ability to compare their performance to those of similar peers within a franchise system while further comparisons to competitors or operators of frequently non-franchised independent businesses (e.g. dry cleaners and the like) may be totally unavailable. Additionally, even small or medium sized Franchisors desire to operate at an “enterprise” level by viewing a comprehensive, rolled up, or consolidated view of their entire business while still maintaining the ability to “drill down” to the individual unit or peer group levels to understand the details of any specific unit, operation or item. Thus most SMB business owners struggle to identify, compute or visualize their own business data let alone find or understand effective business metrics or measurements that are generated for comparative or BI purposes. These SMB users are even further behind when seeking to compare themselves to other businesses as part of an effective SMB BI community using “peer groups” or other BD techniques.
Franchisors have a fiduciary legal obligation to support the franchisees in their efforts to successfully operate the concept. Unfortunately, Franchisors typically struggle to know how their Franchisees are performing as it is very difficult to collect, consolidate and report on the key operational and financial indicators for each of the Franchisees. At least one known reason for this difficulty is because many of these individual Franchisees utilize different operational and financial reporting systems that cannot be easily collected from, consolidated or reported upon due to their different data storage formats, different product versions or non-standardized product deployments. As a result, Franchisors are often left to advise Franchisees on how to improve their business and operational performance with very limited data and they lack the ability to compare them to peer groups and or regional norms within the concept or industry. Additionally, while most businesses and business consultants desire to identify operational Key Performance Indicators (KPI), having limited data makes it difficult to identify and monitor these businesses on a consistent basis.
While Franchising is one example of an industry that can utilize a remote data collection system with a dynamic and adaptive ETL capability, there are many other industries or business models which can benefit from such a system. Other examples include trade associations, co-operatives, or distributors and the like, but can also include branch or field offices of large corporate enterprises. Another example is a bank or credit provider who desires to monitor the financial health of one or more businesses to which they make loans or to whom they extend lines of credit. This enables a new type of loan and lines processing and lines of credit tracking which may be tied to financial measurements such as accounts receivable (AR), cash flow or profitability. Typically, businesses who desire to remotely monitor the financial and or operational parameters of a business depend on emailed or faxed copies of monthly, quarterly or year-end reports which are often lost, ignored or obsolete by the time they are received or reviewed. Worse yet, comparison among and between businesses is difficult if not impossible without normalizing the financial data into consistent sets using a standardized “Chart of Accounts” (COA) to consistently quantify accounting and financial data. In addition, these reports are inadequate to monitor dynamic business conditions and certainly cannot provide monitoring in a near real time and consolidated manner which may reduce business or lending risks without extensive customized Information Technology (IT) systems and support personnel. In general, the problem and challenges of remote data collection and ETL processing can be seen to apply to any and all businesses with multiple locations where financial accounting, POS or LOB applications operate and where the need to monitor, rank or compare these businesses requires access to the data from each location in a consolidated or “rolled up” and optionally a standardized or normalized fashion. Finally, for purposes of definition, readability and clarity, the use of the terms, “cloud system, “Big Data” (BD) system, SaaS system or enterprise websites or portals and the like whenever used in the application should be seen to refer to similar if not operationally or functionally identical concepts or usages or be interchangeable when viewed from the point of view of the invention.